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ABSTRACT 

This report discusses in detail the 198f -89 
evaluation study developed by the University of California at Los 
Angeles Center for Technology Assessment amd the Apple Classrooms of 
Tomorrow (ACOT) administration for the purpose of determining the 
impact of the ACOT project on students, teachers, and parents. Both 
student achievement in writing and student attitudes toward computers 
were measured, together with teacher attitudes eux)ut curricular 
practices, student's achievement, emd occupational stress, amd 
attitudes of parents about the impact of ACOT on their children* 
Measurement instruments on attitudes were primarily questionnaires. A 
norm-referenced achievement test in basic skills and educational 
development was used to establish a basis for comparison, and a study 
of written composition was used to rate writing achievement. Results 
indicate that: (1) there is inconclusive evidence for ACOT 
contributing to students* achievement at a level beyond that of 
conventional instruction; (2) the ACOT project neither undermines 
student interest and motivation, nor enhances aiffective aspects of 
their school experience; (3) in spite of the amount of variability 
among those who responded, it can be determined that ACOT teachers 
experience considerable personal and professional impact; and (4) 
response bias among parents limits the ability to infer ACOT effects 
on parents* views or home activities. Overall satisfaction with the 
children* s progress was reported, and computers appeared to be 
increasingly integrated with home activities. It is concluded that 
assessment of ACOT requires new documentation amd evaluation tools 
capable of measuring the complexities of ACOT effects. (DB) 
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In 1987. an evaluation design was developed by UCLA Center 
for Technology Assessment (Baker U Herman, 1987) in conjunction 
with the ACOT administration. The design is a comprehensive, 
coordinated, multi-component evaluation of the impact of the ACOT 
project on ACOT students, staff, and parents. Key attributes of the 
plan are: 

• collection and analysis of a broad range of potential student 
outcomes; 

• collection and analysis of such information over time: 

• linking outcome data with information on instructional 
process and othei school context variables to provide 
explanatory power for findings: 

• linking multiple indicators of key outcomes to strengthen the 
vahdity of findings: 

• combining the strengths of both quantitative and qualitative 
methodologies: 
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• providing uniform data collection strategies and measures 
across the devise ACOT site:;, but reserving places for 
interests, measures, and effects unique to each site. 

Four basic questions guide the work: 

1. What is the impact of ACOT on students? 

2. What is the impact of ACOT on teachers' practices and 
classroom processes? 

3. What is the impact of ACOT on teachers professionally and 
personally? 

4. What is the impact of ACOT on parents and home life? 

Multiple methods were selected to collect information bearing 
on these questions. Data collection was initiated in the spring of 
1988 with the administration of a range of student outcome 
measures to provide a baseline for comparison with subsequent 
administrations in 1989 and beyond. Student outcomes assessed in 
that initial year included: (1) achievement on standardized tests, (2) 
performance in written composition, and (3) student attitudes. These 
measures are described in the methods section below, and the 1988 
baseline results are described in previous reports (Baker. Herman, & 
Gearhart. 1988; Baker. Herman, «c Gearhart. 1989). Our earlier 
report (Baker. Herman. & Gearhart. 1988) also provides descriptions 
of the ACOT project, the ACOT sites. UCLA's role in ACOT, and the 
evaluation model which organizes our work. 

Our goals for the 1988-89 evaluation were: 

• to complete the process of establishing comparison 
classrooms at all sites 

• to continue tracking the impact of ACOT on students 
with assessments of students' achievement and 
students' motivation 

• to develop and administer a questionnaire to assess the 
impact of ACOT on teachers 



• to develop and administer a questionnaire to assess the 
impact of ACOT on parents 

• to develop procedures for assessing students' growth in 
problem-solving (to be used in 1989-90) 

• to develop a classroom observation instrument (to be 
used in 1989-90) to document changes in classroom 
process 

Throughout 1988-89. our intent was to carry forward the 
essential structure of the 1 988 evaluation plan while reviewing other 
measures for inclusion or substitution. We collaborated with ACOT 
teachers and with the Advanced Development Group of Apple Inc. in 
our efforts to adapt the evaluation to the evolving ACOT context. 
Thus. 

• At two sites, an experimental version of the writing 
assessment was created to permit students to revise their 
essays on a second day. mirroring the writing method some 
students were using in their classroom assignments. 

• At one site, teachers and UCLA staff collaborated in the 
writing of prompts for the writing assessment, so that the 
content of the prompts reflected the kinds of assignments 
that teachers typically assign and that teachers felt would 
result in studen*"' best performance. 

• At most sites teachers provided us information concerning 
the essay genre that received the most emphasis at each 
grade level, and we focused the selection of writing tasks on 
those genres. 

• At the three sites with elementary level grades 2 through 4. 
we revised our methods of assessing children's writing 
competence by adopting an evaluation rubric constructed 
for the elementary grades. The lEA scales used in the prior 
year of our ACOT evaluation had provided us with a national 
comparison sample for Grades 6 through 10. The 
elementary rubric, developed from the same sources as the 



lEA scales and validated by a California school district 
(Conejo Valley), provides us with a district comparison 
sample of Grade 3 students. 

• At one site with primary grade children, the School Attitude 
Measure, normed for grades 4 through 12, was replaced 
with the Self -Concept and Motivation Inventory, normed for 
grades IC-3. At this site. ACOT students in a grade below 4th 
completed both the SCAMIN and SAM, permitting us 
comparison of students' performance on both measures. 

• New questionnaires for teachers were developed and 
distributed at all sites. 

• A standardized measure of occupational stress, the 
Occupational Stress Inventory, was added to the measures of 
teacher effects and distributed to all sites. 

• New questionnaires for parents were developed and 
distributed to all sites. 

Additional' work in classroom observation and in assessment of 
problem-solving was carried out in 1988-89 in preparation for 
continued expansion of the ACOT evaluation in 1989-90. 

• Assessment of changes in classroom processes provide one 
level of explanation for any of the changes documented in 
student outcomes. We piloted both qualitative and 
quantitative procedures for documenting changes in 
classroom process. 

• Teachers across sites commonly report that ACOT students 
are becoming better problem -solvers. In 1988-89, we 
collected a range of information from each site concerning 
teachers' problem- solving curricula and their perceptions of 
students' growth in problem-solving. Information included 
written descriptions of curricula, informal interviews with 
UCLA staff, and videotapes of children's problem -solving 
activities. These background data were the basis for the 
design of problem-solving tasks to be used in an assessment 
of elementary ACOT children's problem-solving competence. 
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Measures 



Student Achievemeiit 

lova Tests . Our evaluation design called for a norm -referenced 
achievement test to be administered in 1987-88 and each 
subsequent year of ACOT evaluation, and the Iowa Tests of Basic 
Skills and the Iowa Tests of Educational Development were selected 
for this purpose. Compared with most other norm-referenced tests, 
we believe the Iowa Tests best allow us to compare students' 
performances across the many grade levels at ACOT. At the 
elementary level, we focused on 4 tests which reflect core emphases 
at most sites: Vocabulary, Reading Comprehension, Mathematics 
Concepts, and Work-Study Skills/Visual Materials. At the 
secondary level, we chose two tests: Vocabulary, and Social Studies. 

Writing . All students responded to writing prompts that asked 
for either narrative, descriptive, peisuasive, or expository writing. 
These prompts were derived from those used in the International 
Association for the Study of Educational Achievement (lEA) Study of 
Written Composition (Baker, 1987). The lEA study included national 
samples of students, but only in Grades 6 and 10. Therefore. ACOT 
students' essays in Grades 3 and up^ (grades close to the grades 
assessed in the lEA study) were rated by specially trained raters 
who used rating scales also employed in the lEA study. These scales 
include ratings on Overall Impression, Organization. Content, and 
Style. (Detailed explanations of this system arc included in Appendix 
Al). The results were compared with ratings for Grades 6 and 10 
U.S. students in the lEA sample. 

The essays of ACOT students in Grades 2-4 were rated with 
scales developed and used by Conejo Valley School District, a 
southern California district that has developed its own rubric 
(derived from the same sources as the lEA scales) for assessing the 
competence of its 3rd graders in narrative writing. Conejo Valley can 
be characterized as middle to upper-middle class in socioeconomic 
status, and we do take into account differences in population 
characteristics when we compare ACOT and Conejo performance. 



* There vas one exception to this procedure. Aton'e site, we collected writing 
samples from 'pre- ACOT' students who were in a grade below 3 but would enter 
the 3th grade in a subsequent year. For these students, we used the lEA scales 
to facilitate following their progress in future years. 



Results for ACOT students were compared with those for Cone jo 
Valley 3rd graders in narrative writing. (Detailed explanations of the 
Cone jo scales are included in Appendix A2.) 

Student Attitudes 

Five approaches were used to assess student attitudes: 1 ) 
responses to a nationally normed measure (School Attitude Measure, 
for grades 4-8; Self -Concept and Motivation Inventory, for grades K- 

2) : 2) ratings of students' attitudes toward computers in their 
persuasive essays on a computer topic; 3) responses of teachers to 
questionnaire and interview items concerning their perceptions of 
students' attitudes; 4) responses of parents to questionnaire and 
interview items concerning their perceptions of students' attitudes; 

3) examination of student attendance and mobility patterns for sites 
that were able to provide us these data. Descriptions of the Teacher 
and Parent Questionnaires are found in later sections. 

Nationally n ormed measures. School Attitude Measure (SAM): 
The SAM is a self -report survey instrument published by Scott- 
Foresman consisting of five scales: Motivation for Schooling; 
Academic Self-Concept, Performance-Based; Academic Self-Concept, 
Reference-Based; Sense of Control; and Instructional Mastery. 
(Details of the five scales are provided in Appendix Bl). SAM is 
normed for grades 4-12 and was administered to all grades within 
this range. Self -Concept and Motivation Inventory (SCAMIN): The 
SCAMIN is a self-report survey instrument published by Person-0- 
Metrics, Inc. consisting of 4 scales at the early elementary level 
^Achievement Needs, Achievement Investment, Role Expectations, 
and Self-Adequacy). (Descriptions of the scales are included in 
Appendix B2.) It is normed lor IC-12 and was administered to all 
Grades below 4. 

Students' attitud es toward computers in persuasive essavs. 
Raters made a judgment that students felt either positively or 
negatively about computer use from the content of their persuasive 
essays on computer topics. 

Archival data — student attendance and mobility . Sites were 
asked to provide UCLA with attendance and mobility data for each 
student for 1988-89, 1987-88, and 1986-87 Compliance with our 
request varied across sites. 



Teachers' Practices, Percetotions. and Attitudes 



Two approaches were used to assess the impact of ACOT on 
teachers: 1) responses to a teacher questionnaire, 2) responses to a 
normed measure of occupational stress (the Occupational Stress 
Inventory). 

Teacher Questionnaire. The Teacher Questionnaire focused on a 
range of topics, including curricular practices, perceptions of 
students' achievement, and perceived stress. (See Appendix Cl for a 
copy of the instrument, and C2 for the scheme used to analyze 
responses to open-ended questions.) At sites 1. 2, 3. and 3. both 
ACOT and (k)mparison teachers received the questionnaires: at site 4. 
only ACOT teachers received them. The rate of return varied 
markedly by site and group. 

Occupational Stress Inventory (OSI). The OSI. published by 
Psychological Assessment Resources, Inc.. is a self-report survey 
consisting of three dimensions of occupational adjustment: 
Occupational Roles, Personal Strain, and Coping Resources. Within 
each domain, there are a number of scales. Occupational Stress 
includes Role Overload, Role Insufficiency. Role Ambiguity. Role 
Boundary, Responsibility, and Physical Environment. Personal Strain 
includes Vocational Strain, Psychological Strain, Interpersonal Strain, 
and Physical Strain. Coping Resources include Recreation. Self-Care. 
Social Support, and Rational/Cognitive Coping. (See Appendix D for 
detailed descriptions of these scales.) The Personal Strain and the 
Coping Resources items were offered to teachers as optional. 

Parents' Views, and Home Us es of Com outers 

A questionnaire was developed to assess the impact of ACOT on 
parents and the home. 

Parent Questionnaire . The Parent Questionnaire focused on a 
range of topics, including parentr' perceptions of the impact of ACOT 
on their children, parental aspirations for their children. ?nd uses of 
the computer at home. (See Appendix El for a copy of the 
instrument, and Appendix E2 for a copy of the scheme developed to 
analyze parents' responses to open-ended questions.) At sites 2, 3. 
and 3. both ACOT and Comparison parents received the 
questionnaires: at sites 1 and 4, only ACOT parents received them. 
The rate of return varied considerably by site and group. 
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Datm Collection Design and Procedures 



Comparison groups . In 1988-89. all sites provided us with a 
comparison group. Two sites provided comparisons within the same 
school, although comparisons at one of these sites were available 
only for 2 of the 3 ACOT grade levels. Two other sites provided 
comparisons from another school, either within the same district or 
in a similar district nearby. The remaining site provided 
comparisons for one grade within the school and comparisons for 
another grade in another school within the district. Not surprisingly, 
the similarity between ACOT and comparison group characteristics 
varied across sites. At some sites, students were selected for ACOT 
by lottery from an applicant pool; since few (if any) of the rejected 
applicants were students in the comparison sample, the comparison 
students were likely to be different in their achievement, motivation, 
and family background. 

Data collection. Data collection instruments and directions for 
their administration were mailed to ACOT prior to the time scheduled 
for administration. Note that this timing for the spring assessments 
pushed some of the data collection uncomfortably close to the end of 
the school year. 

The order of administration for the spring student outcome 
measures was set to minimize the impact of the testing on reported 
attitudes and. to the eitent possible, to provide variety in scheduled 
test activities. The order was: 

1. School Attitude Measure 

2. Writing 

3. Iowa Tests 

UCLA was responsible for ordering test materials and providing 
instructions. Site coordinators were responsible for arranging for 
administration, assuring adherence to requested procedures, and 
returning completed materials to UCLA. In general, regular 
classroom teachers served as test administrators for the writing 
measures, but some sites arranged for a special test proctor for the 
Iowa Tests and ihe attitude measures. Most sites complied in the 
administration of some or all of these measures. The comparison 
groups varied considerably in their willingness or ability to 
cooperate with all components of our study. 

8 
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The Parent and Teacher Questionnaires and the Occupational 
Stress Inventory were distributed by classroom teachers or site 
coordinators in late spring. Some of the ACOT teachers provided 
incentives to students (e.g., special pencils) as a means to encourage 
parent cooperation. Two sites with comparison samples at different 
schools did not distribute the Parent Questionnaire, and one of these 
two sites did not distribute the Teacher Questionnaire or the Stress 
Inventory. As reported in relevant results sections, rates of return 
of these 3 self-report surveys varied considerably by site, grade, and 
group. 

Sites were also asked to provide archival data on district 
achievement test results and absences for ACOT and Comparison 
students for the previous three years. Site compliance for this 
request was less than for the common student, teacher, and parent 
measures. 

Data anaivsis . UCLA prepared all SAM. SCAMIN. and ITBS 
materials for machine scoring by the test publishers. Because most 
ACOT students used word processors to write their essays, all 
untyped essays w:Te typed to assure that raters were unaware of a 
students' status as ACOT or Comparison. UCLA prepared the essays 
for rating by separating Grades 1-4 essays (to be rated with the 
Conejo Valley rubric) from Grades 5-11 essays (to be rated with the 
IE A rubric), deleting students' names, grades, and dates from each 
essay, and scrambling the essays within the lEA and Conejo 
groupings so that raters were 'blind* with respect to school, student s 
gender, grade, or date of completion. Responses to the close-ended 
items on the Teacher Questionnaire, Parent Questionnaire, and Stress 
Inventory were key-punched for quantitative analysis. Responses to 
the open-ended items were coded with schemes for qualitative 
analysis constructed from the responses themselves. 

Results 

Evaluation of the ACOT project is made difficult by the 
extraordinary diversity among its five sites in student 
characteristics, curriculum emphases and activities, parent and home 
characteristics, and nature of the comparison samples. In addition, 
district restrictions, scheduling conflicts, and site willingness 
confounded the administration of each of our measures across all 
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sites to both ACOT and Comparison groups, and leaving some gaps in 
our results. 

In our results, sites are identified by number (1-3) rather than 
by school or district name. Because grade level and site are 
confounded, the letters A, B, and C are used to signify grades at a 
site. These letters are intended only to reflect lower (A) and higher 
(B or C) grade levels, and do not correspond across sites. For 
instance, at one site 9th grade would be labeled A and 10th grade 
labeled B, while at another site 1st grade is labeled A and 2nd grade 
labeled B. While somewhat more difHcult to read, these decisions are 
consequences of our agreements to protect the privacy of the 
teachers and students at our sites. 

All results reported below have excluded special education 
students, with the exception of Grade A at Site 2, a transitional' class 
of special education students not yet ready for Grade B at that site. 
Note as well that the letters used to designate grade levels in this 
1988-89 report do not necessarily correspond to the same levels 
indicated in the 1987-88 report. Thus at site 2, for example. Grade B 
is in fact the same as Grade A in the 1987-88 report. These changes 
in grade identification resulted from either reorganizations of the 
ACOT project at certain sites or extension of our data collection to 
additional ACOV grades. Comparisons of last year s and this year's 
student results (using this year's indexing system at all times) are 
included in this report to minimize any confusion in comparing 
results across the two reports. 

Grade levels sometimes reflect years in ACOT and sometimes 
do not. At site 1, almost all students in Grades A through C were in 
their first year of ACOT in 1988-89 (students were chosen by lottery 
for admission to ACOT in Grades A and B, but all students in Grade C 
were ACOT). At site 2, students are in ACOT for only one year, either 
Grade A (special education students) or Grade B. At site 3, tlie ACOT 
project expanded in 1988-89 to include the next lower grade, and 
thus students in both Grades A and B were new to ACOT. and 
students in Grade C were in their second year. Only at sites 4 and 3 
does each successive grade indicate one more year in ACOT. 

Student Achievement 

Iowa Tests of Basic Skills . Sites 2, 3, 4, and 5 administered the 
Iowa Tests. The results for the Iowa Tests were examined in three 
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ways: group results for 1988-89 (percentiles based on mean grade 
equivalents) for each subtest (Appendix F). group comparisons of the 
1988 ramples with the 1989 samples (Appendix G). and repeated 
measures comparisons of students who took a test on two occasions 
(Appendix H). 

A percentile score of 50 is the median of the national sample, 
the point below which half the national group fell and the so-called 
national average. Because percentile ranks are generated for each 
grade level, the reported scores control for age and grade level 
differences across sites. While 3 of the 4 cooperating ACOT groups 
are performing above the national norm (disregarding the special 
education Grade A at site 2). the data (Tables Fl and F2) verify the 
great differences among the sites in overall student academic 
achievement. 

At three sites with Comparison results. ACOT students scores 
were generally higher at each cycle of administration. However, 
most of these results appear to reflect pre-existing and fairly stable 
differences between ACOT and Comparison groups, with the possible 
exception of Site 4. 

• Based on 1988-89 group performance (Table F3). ACOT 
students are generally performing more competently on the 
Iowa Tests than the Comparison students. At site 2 there was 
a decline in ACOT percentile scores from fall to spring, 
particularly lor Grade B, a possible reflection of exceptional 
stress in the spring when the building was undergoing 
extensive construction at the ACOT wing: nevertheless, the 
ACOT grade B students were still performing markedly higher 
than Comparison students. 

• If we compare 1988 scores with 1989 scores (Tables Gl and 
G2), ACOT students at site 3 have tended to score higher than 
their Comparisons both years. At site 4. comparisons between 
groups is made difficult by changes m selection of the 
Comparison sample. In 1987-88, the Comparison sample was 
an entire school district; in 1988-89, only volunteer 
classrooms in that district participated (with few students in 
Grade A). Results indicate that at both grade levels the 
differences favoring the Comparison sample in 1987-88 were 
reduced or even reversed in 1988-89, a promising finding. 




However, sampling differences (especially for Grade A) could 
also be an explanatory factor, making interpretation difficult. 

• If we compare ACOT and Comparison students' grade 
equivalent scores for students who took the ITBS on two 
occasions (Table HI), the evidence suggests that differences 
in AOOT and Comparison classrooms are due to pre-esisting 
and stable differences between the two groups. At sites 2 
tnd 3. the growth of both groups at each grade for each test 
was often weak (less than the expected growth of .7 years for 
fall-spring and 1.0 years for spring-spring.) At site A, the 
growth of the Comparison students for Grade B tended to be 
greater than ACOT growth. (The ITED does not provide grade 
equivalents, but based on sroup performance. Grade A 
students declined marginally from fall to spring (Table F3). 
and Grade B tnd C students (comparing them with their 
results as Grade A and B students last year. Table F2) also 
declined somewhat from 1988 to 1989.) 

There is little consistent evidence that students with more 
years of ACOT experience were performing better than students in 
their first year (Tables Gl and G2). 



• At site 3. the relative performance of Grades A and B (both 
first year ACOT) vs. Grade C (second year ACOT) shifted from 
test to test. At site 4. ACOT Grades A and B performed 
similarly (although Comparison Grade B performed somewhat 
better than Comparison Grade A). At site 5. Grade B tenrica to 
score higher than either Grade A or .. 



Local district assessments Results from district assessments 
provide additional evidence of pre-existing ACOT and Comparison 
differences (Tables 11-13). 



• District testing results (Appendix I) were provided by three 
sites. At Site 1 (Table II). Comparison students for Grade B 
scored slightly higher than ACOT students on several SRA 
subtests. The Comparison students for Grade B at Site 2 
(Table 12). consistent with the ITBS results at this site, scored 
markedly lower than ACOT students on Stanford tests. At Site 
3 (Table 13). Comparison students have scored considerably 
lower than ACOT students on the Comprehensive Tests of 
Basic Skills. 
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Writing . We had several goals for our evaluation o^ students* 
writing competence: (1) follow students' growth from 1988 to 1989; 
(2) focus on assessment of th? genres most emphasized at each site: 
^3) add an experimental version of the assessment that permitted 
students to rewrite their essays on a second day, to provide us both 
with an indei of students' competence with the process of revision 
and with a more valid picture of writing ability. To accomplish goals 
(1) and (2). a* each cycle of data collection (fall, winter, and spring of 
1988-89). students were assigned to write on particular genres; no 
student, however, wrote on exactly the same topic more than once 
Note also that when we assessed students' performance in more than 
one genre at any given time, we administered each genre to a 
sampling of students to reduce the number of different essays a 
student would have to complete. To accomplish our third goal, two 
sites agreed to administer first and second drafts in the winter, and 
one of these sites repeated this procedure in the spring. 

All student essays were scored by one of seven raters. The 
Conejo Valley rubric 6-point scales were used for essays in Grades 2- 
4. The lEA 5-point scales were used for essays in Grades 5-11. 
(Note that at site 3 this procedure resulted in Conejo scoring for 
Grade A and lEA scoring for Grades B and C.) For both sets of ratings, 
reliability of the scoring process was assessed by computing the 
agreement among all combinations of raters for at least 20% of the 
essays in each genre. (When essays were double-scored for 
reliability, the score used in data analyses was the average of the 
two raters' judgments.) Interrater agreement (Tables Jl and J2) was 
high for both scales. Inter correlations among the scales composing 
each rubric (Tables Kl and K2) were high, and we are therefore 
reporting only the results for General Competence/Cone jo and Overall 
Impression/IEA. For essays that included the second draft option, 
we used only the first draft rating in our primary analyses. 

The results for the writing assessment were examined in four 
ways: group results for 1988-89 (mean ratings. Appendix L), group 
comparisons of the 1988 samples with the 1989 samples (Appendix 
M). repeated measures comparisons of students who wrote on a 
genre on two occasions (Appendix N). comparisons of first and second 
draft ratings (Appendix 0). We should stress that the amount of 
data collected was considerable and the number of variables 
examined was large (genre, time, grade, group), and therefore the 
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findings below of occasional differences in the context of the sheer 
number of comparisons should not be overinterpreted. 

On the Conejo 6-point scales, a rating between 3 and 4 indicates 
competent performance: on the lEA 3-point scales, a ratir;"- of 3 
indicates competence. Based on these criteria, the writing o' many 
ACX)T students is weak, although the results show a considerable 
range in students' performance across sites. 

• For essays rated on the Conejo scale {1988-89 ACOT 
performance. Table LI), only students '.n Grade C at site 1 and 
Grade B at site 2 received mean ratings above 3 for 
descriptive (both sites) and persuasive (Site 2) writing. For 
those scored with the lEA scales (Table L2), only students at 
Site 5 i -aived mean lEA ratings at 3 or above. 

• There is. however, some evidence of growth with grade level 
(and years of ACOT experience) at Site 3 and Site 3 

• ACOT students did occasionally perform as well as the 
national norm group provided by the lEA Grade 6 and 10 
students' and the upper SES comparison group provided by 
Conejo Valley School District {1988-89 ACOT performance. 
Tables L3 and L4). Although we have only ratings of Conejo 
Grade 3 on narrative, our adaptations of the rubric for 
descriptive writing indicated that the Grade 3 students at Site 
1 were writing almost as well as the Conejo students might 
have in this genre (Table L3)- This is not a surprising result, 
in that the Site 1 population is more similar to Conejo than is 
Site 2 or 3. The persuasive essays of this Grade in the spring 
were also rated at a comparable level (Table LI). Grade 6 
students at Site 4 were writing descriptive essays comparable 
to the lEA sample, and persuasive essays more competently 
than the lEA students. Grade 10 students at Site 3 were 
writing roughiy comparable expository essays.^ At various 
times during the year, students' writing in other genres 
equaled or exceeded lEA means (Table L2). 



' At this site, hovever, teachers participated in the writing of new prompts 
that better matched their instruction in all genres. These prompts vere 
administered in the vinter and spring of 1989, and ve cannot determine to 
vhat degree teacher-selerted prompt content may have supported the quality 
of students' vriting. 
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Interestingly, the strengths that students displayed were not 
consistently those that sites indicated were emphasized at each grade 
level. 

• We examined the results across genres for the spring 
outcome essays. The relatively even performance of students 
at Sites 2 and 4 is consistent with site descriptions of their 
curriculum (Tables LI and L2). Site 3 reported expository 
emphasis for Grades B and C; since persuasive and expository 
essays are structured and assessed very similarly, the 
strength of the persuasive ratings is also consistent with site 
curriculum descriptions. However, descriptive and 
persuasive ratings are often higher than narrative for sites 
that reported narrative emphasis (Site 1, Site 3/Grade A, Site 
5/Grade A). 

These discrepancies could have several explanations, among them: 
mismatche5 between our and teachers' definitions of genre; the need 
for more focused instructional efforts; differences in prompt 
difficulty. 

The writing results support again a finding of pre-existing and 
fairly stable differences between ACOT and Comparison groups, with 
ACOT -Comparison differences generally identical to those found for 
the Iowa and district achievement tests. 

• Analyses of 1988-89 group performance (Conejo/Table L5: 
lEA /Table L6) tend to show ACOT students performing better 
than Comparison students. Only Site 1 shows a possible 
contribution of the ACOT experience to students' writing 
competence. 

The 1989 ACOT students tended to perform more competently 
than the 1988 sample at the same grade level.^ 



^ Comparisons of last year's and this year's ratings for Grades 2-4 cannot be 
direct, since last year's essays were rated only with lEA scales. However 
results from an equating study (double-scoring with both the Conejo and lEA 
schemes) using a sample o^ the 1989 essays indicated very high 
correspondence between ratings on these scales, with 6/3 of an lEA rating 
equivalent to a Conejo rating. Therefore, to compare the 1988 and 1989 essays 
for two elementary grades that had also been assessed last year ve multiplied 
the 1988 lEA means by 6/3. 




• Analyses of group comparisons of the 1 988 samples with the 
1989 samples indicated that students at sites 1 and 2 may 
have been writing more competently than their same-grade 
peers last year (Table Ml). Similarly, the lEA ratings of 
students at Sites 3. 4, and 3 were higher at each grade level in 
1989 than they were in 1988 (Table M2). 

Although we have Comparison results only for Site 3. the essays of 
the Comparison students in Grades B and C were also rated more 
highly in 1989 than in 1988. This result casts doubt on any specific 
effect of the ACOT experience, and suggests the most conservative 
interpretation that provision of our tasks and rating schemes has 
been having an effect on writing instruction for both groups at all 
sites. 

At most sites, students in both the ACOT and Comparison 
groups tended to perform as well or better over time. 

• Analyses of repeated measures comparisons of students who 
wrote on a genre on two occasions reveal inconsistent trends 
across genres, sites, and grades (Tables N1-N6). Few changes 
were significant by statistical criteria. 

As for the group performance results, these analyses suggest that our 
writing assessment is contributing to students' writing growth in 
both ACOT ana Comparison groups. 

Some students were able to improve the quality of their 
writing in their second drafts. 

• Students' second drafts were generally rated more highly 
than their first drafts at Site 3 (the secondary site) (Tables 01 
and 02). 

St udents' achievement: Summary and Interoretation. None of 
results for the achievement assessments indicates that ACOT is 
contributing to students' growth at a level beyond that of instruction 
in more conventional contexts. Differencies between ACOT and 
Comparison students tended to be consistent across measures and 
time, indicating pre-existing and fairly stable differences between 
groups more than any direct effect of the ACOT experience. 
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Although many ACOT students were performing at or above 
the national median on the Iowa Tests and their district assessments, 
there was considerable variability across sites reflecting typical 
patterns of performance and growth in schools with comparable 
population characteristics. 

Many ACOT students were not writing at the levels of 
competence defined by our assessment rubrics in all genres, but 
there was evidence of growth in that the 1988-89 students showed 
improvement over time. Both the ACOT and the Comparison 1988- 
89 samples were performing above the 1987-88 samples, a pattern 
which may indicate that our writing assessment itself was serving as 
an effective intervention. At several sites, selected grades were 
sometimes performing as well or better on some genres than our 
comparison Conejo and lEA samples. The writing data gave us a 
quite differentiated picture of students' growth over time and with 
grade level as well as relative performance among genres. However, 
once again the evidence best supported a finding of pre-existing and 
stable differences between ACOT and Comparison groups, and not an 
effect of ACOT. At only one site were there results suggesting that 
ACOT students may have been improving at a rate greater than their 
comparison peers. 

Our experiment with assessment of writing process, through 
comparative ratings of first and second drafts, produced most 
provocative though preliminary results suggesting that secondary 
level students are more able to improve the quality of their essays in 
their rewrites. More systematic administration of this version of our 
assessment is needed. 

Student Attitudes 

In this section we report results from 2 normed instruments, 
the SAM (Appendix P) and Lli (Appendix Q), ratings of 

students' attitudes toward compute, in their persuasive essays on a 
computer topic (Appendix R), and patterns of students' attendance 
(Appendix S). Teachers' and parents' reports of students' attitudes 
are presented in sections on teacher and parent effects. 

SAM . The results for the SAM were examined in three ways: 
group results for 1988-89 (percentiles based on mean weighted raw 
scores) for each subtest, group comparisons of the 1988 samples with 
the 1989 samples, and repeated measures comparisons of students 

ERIC , I u 



who took the test on two occasions. Problems with site compliance 
and errors in administration interfered with a number of these 
analyses. 

Student responses to the SAM are reported in terms of national 
percentile scores, where 30 represents the score point below which 
half of the national sample fell. Most students' attitudes were 
approximately equivalent to or more positive than the national 
average. 

• Analyses of 1988-89 group perfortr^nce (Table PI) indicate 
some diversity among ACOT students across sites. The 
somewhat lower scores of the special education class (Grade 
A) at Site 2 are not surprising given the population. 

By statistical criteria, there were almost no differences 
between ACOT and Comparison students' attitudes at four sites. 

• However, consistent with the achievement results, ACOT 
students in Grade B at Site 2 tended to report more positive 
attitudes than their Comparisons (Table P2). At site 3, Grade 
B ACOT 'students reported more positive attitudes than their 
Comparisons; since ACOT students' attitudes did not differ 
across grade levels, this result reflects the rather negative 
attitude of Grade B Comparison students and not an effect of 
ACOT. 

The 1988-89 students did not appear to differ in their 
attitudes from 1987-88 students. 

• Group comparisons of the attitudes for ACOT students in the 
same grades in the spring of 1988 and m 1989 (Table P3) do 
not show consistent patterns across the four sites for which 
we have these comparisons. The attitudes across most 
grades at each site were similar. 

The 1988-89 students did not appear to have changed in their 
attitudes since the spring of 1988. 

• Consistent with the group results, repeated measures 
analyses of the attitudes of ACOT students who took the SAM 
on two occasions (Table P4) found few statistically significant 
changes over time, and these were small in size. 
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SCAMIN. The SCAMIN was administered only at Site 1, where 
students' grade level rendered the SAM (normed for grades 4-12) 
inappropriate. 

• Results from the SCAMIN for students at Site 1 (Table Ql) 
indicated somewhat more negative attitudes than any of our 
results for the SAM at any site. Nevertheless, most scales of 
the SAM were correlated with most scales of the SCAMIN for 
the small sample of students who were administered both 
assessments (Table Q2). suggesting that both instruments 
were assessing similar affective domains. 

• The only SCAMIN scale which did not correlate with the SAM, 
Achievement Investment, also produced the only difference 
between the ACOT and Comparison students at Site 1. This 
scale is intended to indicate students' concern to avoid 
consequences of failure in school. 

Students' attitudes toward comnuter use in persuasive essavs 5. 
Both ACOT and Comparison students at all sites expressed very 
positive opinions of computer use in their essays. 

• The majority of students in all groups across sites wrote 
favorably about computer use in schools (Table Rl). 

Attendance . Table SI contains patterns of student attendance 
from 3 sites. Results across sites and grades are inconsister. . 

• At Site 1. attendance was somewhat less for ACOT than 
Comparison students, but at sites 2, 4, and 3. attendance was 
a bit greater for ACOT students. These differences at Site 5 
have been consistent over time, reflecting probable sample 
differences rather than an ACOT effect. 

Student at titudes: Summarv. Based on results from normed 
attitude measures, content analyses of students' essays of computer 
use. and patterns of student attendance, the ACOT project is neither 
undermining students' interest and motivation, nor is it enhancing 
the affective aspects of their school experience beyond that of their 



' Interrater agreement, based on double-scoring of 20% of ail persuasive 
essays, was 96% . 
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peers. It is of note that ACOT students do not appear to be tiring of 
their interest in the innovation. Most ACOT and Comparison students 
reported generally positive attitudes toward schooling and toward 
computer use. 

Teacher Effects 

In this section we report the results of two instruments, the 
Teacher Questionnaire and the Occupational Stress Inveiitory. 

Teacher Ouestionnaire . Our efforts at analysis were limited by 
an uneven r&te of return across sites and groups (Appendix T, Table 
VI). The number of teachers responding to the questionnaire was 
not high, and therefore we combined teachers across sites, separating 
them by lower elementary (grades 1-3). ' pper elementary (grades 
4-6), and secondary (grades 9-12). Still tne number responding was 
low for certain combinations of levels and groups: there are few 
comparison teachers for lover elementary and secondary levels, and 
at the upper elementary level, most of the ACOT teachers are from a 
different site from most of the Comparison teachers. We therefore 
avoid making comparisons between ACOT and Comparison teachers. 

Below our analyses are divided into topical areas covered in 
the questionnaire. Ai^alyses of the questions administered to all 
teachers are contained in Appendix U. Analyses of the responses to 
the open-ended questions administered only to ACOT teachers are 
contained in Appendix V. One rater scored all open-ended 
responses using the schemes detailed in Appendix C; to establish 
reliability, a second rater scored 20X of the sample distributed across 
sites, and groups. Agreement was excellent: 100% for responses to 
all questions with the exception of 83% for teachers' new 
expeaations for students. 

Curriculum emphases : Reported curricular time allocations 
showed great variability across grade and group (Table Ul). We 
question the uniformity of teachers' interpretations of these items. 
We also note that certain subject categories (e.g., 'Reading') fit 
elementary level curriculum better than secondary curriculum. 

Nature of classroom assignments. Based on teachers' reports, 
the Oiost common kind of classroom assignment for teachers at all 
levels was a brief, one-day assignment (Tables U2-U4), although 
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secondary teachers, consistent with the advancing matuf ity of their 
students, reported assigning these somewhat less often. 

• Correspondingly, secondary teachers assigned three-day 
assignments in more subject areas more frequently than 
elementary teachers. Note however that elementary teacherr. 
assigned 3-day writing assignments fairly frequently (2-4 
times per month), and upper elementary teachers also 
assigned 3-day reading assignments this frequently. All 
other assignments were much less frequent at every level. 
Not surprisingly, it was in science that students were 
relatively more often engaged in projects and exhibits. 

Student role in assignment cnmoletion Most teachers tended 
to report that they assigned students specific independent work 
more often than providing choice (Table U3). 

• However, students were given some choices with reasonable 
frequency, at least in some subject areas. Students were 
fairly often given choices among several assignments, 
particularly in writing, and were also given choices of the 
order in which they could complete assignments. 

Classroom erouoines . Typical of most elementary instruction. 
ACOT elementary teachers reported organizing an average of 2 or 3 
reading and math groups in their classrooms (Table U6). Students 
were reported to work together in pairs or cooperative groups with 
moderate frequency and to assist one another at all levels, although 
the subject areas organized for peer involvement varied across 
levels. 

Homework . Teachers at all levels reported that they assign 
homework and that most students complete it (Table U7). 

Teachers' sa tisfaction with students' academic and 
socioemot ional orosress . Most teachers were quite f.atisfied with 
their students' academic 9nd socioemotional progress (Table U8). 

• The ratings of the lower elementary teachers tended to be 
the highest across all areas, and the ratings of the secondary 
teachers tended to be the lowest across all academic areas 
except computer skills. Most teachers were quite satisfied 
with their students' attitudes toward school and self- 
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confidence. Upper elementary teachers expressed mild 
concerns about their students' independence and ccoperative 
skills. 

Teachers' iob-related srowth and stress Most teachers 
reported experiencing challenge and growth with great frequency 
(Table U9). Teachers' perceptions of support varied, with a tendency 
to feel more support and recognition from their team teachers than 
from other teachers or principals. 

• Most teachers reported experiencing personal stress from 
their work occasionally or often, although the frequency of 
reported stress was somewhat less than the frequency of 
reported challenge and growth. 

ACOT - a c ontext for curricular revisions:. In optional, open- 
ended questions, some ACOT teachers reported developing higher 
expectations for students since ACOT (Table VI). particularly for 
greater student achievement and for acquisition of computer skills. 

• Teachers at the secondary level were most likely to report 
enhanced expectations for students' abilities to participate 
responsibly as a result of ACOT experience. Some ACOT 
teachers remarked on a variety of new ways of using 
computers for instruction (Table V2). including uses of 
applications, ways of integrating computer u:e into specific 
subject areas, uses of computers for encouraging cooperative 
work, and (infrequently) individualization of curriculum. 

ACOT teachers' views of the effects of ACOT on themselves . 
ACOT teachers were asked about the positive (Table V3) and the 
negative (Table V4) impact of the ACOT experience on them. 
Respondents took the opportunity to describe a wide range of effects. 
Teachers often remarked on the value of computers in their teaching 
and the positive impact on their job interest and performance. 

• TeacbTs at higher levels were more apt to mention benefits 
for improved teacher and student roles. Teachers at the 
upper elementary level were most apt to mention positive 
effects on their feelings of self -worth and their appreciation 
of the opportunities to travel and to meet new people. But 
teachers also expressed concerns. Some expressed worries 
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about curriculum coverage, and many reported stress and 
demands on personal time. 

Questionnaire wrao-UD - ACOT teacher's views of the 
strengths and weaknesses of ACOT . ACOT teachers were also asked 
about the strengths (Table V3) and the weaknesses (Table V6) of the 
ACOT project. Their responses highlight some of teachers' most 
salient feelings about ACOT. In descri^ ing strengths of the ACOT 
project, many teachers remarked on ACOT colleague relationships. 

• Additional strengths mentioned varied somewhat across 
levels, with a lower elementary focus on student affect and an 
upper elementary/secondary focus on student-teacher roles. 

In des^^'ibing weaknesses of ACOT, teachers focused most often on 
unproductive staff relationships and on concerns for curriculum 
coverage. 

Occupational Stress Invenlorv (Appendix W, Table Wl). We 
could not break down the Occupational Stress Inventory results by 
site or by elementary/secondary levels, because the scales are 
normed separately for men and women, and therefore the number of 
respondents in any category for either gender would be too low for 
meaningful analysis. Very few teachers completed the inventory in 
any case, and most who did completed only the requested 
Occupational Roles section. 

Note that the scales clustered under Occupational Roles and 
Personal Strain are interpreted differently from those under 
Personal Resources. A higher percentile within the first two clusters 
indicates greater perceived stress; a higher percentile within the last 
cluster indicates perception of greater personal resources. 

The ACOT teachers who responded to the OSI reported stress 
comparable to many adults (i.e., around the 30th percentile 
compared with a sample of adults drawn from a wide range of 
occupations), with the exception of relatively higher stress reported 
for Role Overload, and Physical Environment. Items in the Role 
Overload scale indicate heavy workload with feelings of insufficient 
support or training to meet the work demands. Items in the Physical 
Environment scale indicate environmental interference (e.g.. noise, 
light, heat/cold), erratic work schedules, or isolation. 
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Teacher^:' Summary and Commentary. The distribution of 
Teacher Questionnaire and OSI returns greatly limited the inferences 
we could draw from the results. In particular, we were rot 
comfortable comparing ACOT teachers with their Comparison 
colleagues. In addition. 1988-89 was the initial year for collection of 
these data, and we are therefore unable to use analyses of changes 
over time to help clarify the source of any ACOT-Comparison 
differences. These limitations make it virtually impossible for us to 
identify ACOT effects on either classroom practices or teachers' 
experience and attitudes. 

ACOT teachers' reports of their classroom practices indicated, 
for each level, fairly typical subject matter emphases, classroom 
organizations, and student roles. We acknowledge, however, that our 
questionnaire items, most of which requested either ratings (e.g., of 
frequency) or numerical assessments (e.g.« of time) may not have 
been sensitive to other subtle changes in ACOT classroom practices. 
ACOT teachers appeared fairly satisfied with their students' progress, 
although secondary teachers tended to express less satisfaction than 
elementary teachers. Curriculum becomes more differentiated and 
complex with advancing grade levels, and it may be that the 
necessary elaboration of teachers' objectives results in more acutely 
examined progress among their students. Results from all other 
measures indicated that ACOT has had considerable personal and 
professional impact on teachers. Although all teachers (including 
Comparison teachers) appeared to feel challenged and growing, as 
well as somewhat stressed by the demands of their job, these effects 
seemed more marked for the ACOT teachers. ACOT teachers 
remarked on a variety of benefits of the ACOT experience, for 
themselves as professionals and for their students, and appeared to 
be constructing new interpretations of their own and their students' 
abilities. Secondary ACOT teachers, in particular, consistent with the 
relatively advanced levels of competence of their students, appeared 
to be most actively revising their notions of students' roles in their 
own learning. 

EfTects on Parents and the Home 

This section contains the results from the Parent Questionnaires 
as well as teachers' descriptions of parent involvement reported in 
the Teacher Questionnaire. 




Parent Questionnaires . Appendix X (Table XI) contains the 
rates of return for sites and groups that distributed the Parent 
Ouestio.->'iaires. Parent compliance varied considerably across 
grades, groups, and sites. 

Below our analyses are divided into topical areas covered in 
the questionnaire. Results from the close-ended questions are 
contained in Appendix Y. Results from open-ended questions 
administered only to ACOT teachers are contained in Appendix Z. 
One rater scored all open-ended responses using the schemes 
detailed in Appendix E; to establish reliability, a second rater scored 
20% distributed across sites and groups. Agreement was excellent: 
96% for Child Benefits, 98% for Child Disadvantages, 90% for Parent 
Benefits, and 100% for Parent Disadvantages. 

Background. Family cnaracteristics: Some of the respondents 
were willing to share some background information about 
themselves (Table YD. The results make clear the diversity among 
the sites in characteristics known to be related to student 
achievement, including parents' educational and occupational levels. 

• Consistent with our own perceptions of the sites, the indices 
were quite high at site 1 and quite low at site 3. with other 
sites ranging in between. 

This pattern of site difference in parents' reports was evident in 
many of the the analyses. 

Parents' aspirations. Most of the parents who responded to 
the questionnaire hoped that their children would acquire some level 
of post-secondary education and expected that to happen (Table Y2). 
Similarly, most parents expected their children to work as skilled 
workers or as managers. There was some variation across sites and 
groups. 

• Both parents' aspirations and expectations tended to be 
consistently high at Site 1, where levels of parental education, 
occupation, and income are relatively high. Where we have 
comparison results, ACOT parents' expectations both for their 
children's education and career tended to be higher (with the 
exception of Site 3. Grade B), although the number of 
comparison respondents is low in most cases. 
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Parents' involvement. Parents across sites, grades, and groups 
reported a low to moderate level 0** involvement with their children's 
school (Table Y3). 

• At Site 1. involvement was somewhat greater at earlier 
grades, a pattern typical in the primary grades. At most sites, 
parents of children in older grades reported greater 
knowledge of the school than parents of children in younger 
grades. Teachers' reports of p&rental involvement (Table Y4) 
were generally consistent with parents' reports. The results 
suggest greater attendance at school functions by ACOT than 
Comparison parents, although the apparent ACOT -Comparison 
difference at the Upper Elementary Level must be 
disregarded, since most of the teachers in the ACOT group 
were from a different site from most of the teachers in the 
Comparison group. 

Parents' satisfaction with their children's teacher. Most parents 
were highly satisfied with their children's relationship with their 
teacher (Table Y3). 

• Parents at Sites 4 and 3 tended to report lower levels of 
satisfaction than parents at the other sites. Parents of upper 
elementary and secondary students may have developed 
more differentiated goals for their children's education, and 
therefore more critical views, than parents of younger 
children. ACOT parents at Site 3 may be less satisfied with 
the extent to which the teachers 'press' their students to 
achieve (although the rate of return at this site was often 
low). 

Parents' satisfaction with their children's academic and 
socioemotional progress. Parents of ACOT children from all sites 
tended to report moderate to high levels of satisfaction with their 
children's progress (Table Y6). Progress in computer skills was 
generally very highly rated, but sites (and grades within sites) 
varied in relative ratings across subject areas. 

• At Site 1. parents' satisfaction in most areas tended to decline 
with grade level. At site 3, parents of students in Grade C 
(their second ACOT year) tended to be less satisfied in most 
areis than parents of students ir Jrades A and 6 (first ACOT 




year). The ratings of students' socioemotional progress 
tended to be lover st Site 4 than at other sites. 

Interpretation of differences between ACOT and Comparison 
parents' satisfaction was limited by the small number of comparison 
respondents. 

• Comparisons are available only at Sites ? and 3 (Table Y7). 
(Although comparison parents at Site 3 returned the 
questionnaire, almost none of them completed these items.) 
At Site 2, ACOT parents reported greater satisfaction with 
their children's academic progress in several areas. The same 
trend appeared at site 3 in the results for Grades A and B 
(first year ACOT). but. in contrast. Grade C ACOT parents were 
generally less satisfied than the Comparison parents. 

Parents' views of their children's interest in school at home- 
Table Y8 contains ACOT parents' reports of the frequency with which 
children engaged in. or talked about, school activities at home (lower 
numbers indicate greater freouencv). Parents generally reported 
that their children were showing some interest in school activities 
once or twice a week, although children expressed those interests in 
different ways at different sites and different grade levels. 

• Certain grade level differences (homework and use of school 
software for ACOT students) r fleet the increase of assigned 
homework at higher grade levels. Where there are 
comparison results, ACOT parents tended to report more 
frequent evidence of school activities at home than did 
comparison parents (except at Site 3. Grade C). This result is 
not surprising, since the presence of an identical computer 
both at home and at school would be exnected to increase the 
possibility and therefore the likelihood of school-like home 
activities. 

Parents' reports of children's home activities. According to 
parents' reports (Table Y9), children across sites, grades, and groups 
spend their time in somewhat different but predictable ways. 

• Television watching is common among all children, but less 
common for Site 1 (primary grades, high socioeconomic level) 
and Site 3, Grade C (upper secondary level, with increasing 
demands on time from homework and part-time jobs). 
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• As expected, homework time tended to increase with grade 
level, and of course ACOT children used home computers 
more than Comparison children. 

• Reported patterns of children's reading were not clearly 
related to site, grade, or group. 

• At site 3. there was a suggestion that ACOT children in Grades 
B and C may have watched less TV. played less team sports, 
played outside less, and visited with friends less. If these 
results are valid, they may reflect pre-existing differences 
between groups rather than an effect of ACOT. 

ACOT parents' rep<x'ts of home computer use — Hours per day. 
ACOT children were the most frequent users of the home computer 
(Table YIO). 

• Children at Sites 3 and 5 appeared to spend the most time at 
the computer. The lower socioeconomic level of Site 3 is 
likely to have contributed to the novelty and therefore more 
frequent use of the home computer: Site 5 (secondary level) 
students were assigned homework on the computer nightly. 

• Parents reported quite frequent use of the home computer 
by other family members as well. Parents of secondary level 
students (Site 3) were the least likely to use the computer. 
We cannot determine from this result whether the homework 
demands on Site 3 students interfered with adult use. or 
whether other dynamics in these adolescents' homes resulted 
in less interest or access for parents. 

• Siblings and friends of Grade B. Site 3 children were the most 
likely to use the computer, reflecting again the lower 
socioeconomic level at that site and the inviting interest that a 
new home computer must have provided the other children 
in the ACOT's family and neighborhood. 

Changes in uses of the ACOT computer over time. At most sites, 
parents reported that over time the home computers were being 
used more hours per week and for more uses, particularly for ACOT 
children, suggesting that a novelty' effect was not common (Table 
YIO). 
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Uses for the ACOT home computer. The pattern of uses for the 
coiTkputers revealed differences in students' interests and 
competences across sites, grades, and groups. Common uses for ACOT 
chil'Jren (Table Yll) were games, homework, practicing, personal 
writing, and graphics. (We are not certain that all parents gave 
'programming' the same definition.) 

• There was a greater variety of children's uses reported by 
parents at Sites 3, 4, and 3 (upper elementary and secondary 
levels) than Sites 1 and 2 (primary and middle elementary 
levels). 

Reported uses by ACOT children's siblings and friends (Table Y12) 
were spread more thinly across categories at most sites, but there 
was evidence that it was not uncommon for siblings and friends to 
use the computer for personal needs such as homework and writing, 
as well as for recreation in games and graphics. 

• The finding that siblings and friends were using the home 
computer to do homework at Sites 1. 2, and 4 may reflect the 
greater likelihood that the ACOT high school students (Site 3) 
were using the computer during typical homework times. 

Parents' reports of their own uses of the computer (Table Y13) were 
quite diverse. The more commonly reported uses included personal 
writing, practicing, finances, and games. 

• Parents at Site 1 tended to use the computer for more 
purposes than parents at other sites, while parents at Site 3 
tended to use the computer infrequently and for fewer 
purposes. 

ACOT children's assistance v/ith home computer use. Some 
parents reported that their ACOT children had provided computer 
assistance to them (Table Y14). The frequency reported was not 
high, ranging on average from less than once a month to 2/3 times 
per month. 

• Assistance was most frequent at Site 3. where parents would 
be the least likely to have had prior exposure to computers. 
Children provided assistance in a range of contexts, from 
games to word processing to graphics. Children in older 
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grades tended to be more likely to help their parents with 
more specified applications such as spreadsheets and 
databases. 

ACOT parents' views of the effects of ACOT on their children. 
Because the distribution of parents who completed the open-ended 
questions was very uneven across grade levels, we collapsed 
analyses of open-ended responses within sites across grades. In 
their responses to open-ended questions (Table ZD. parents were 
more likely to comment on the benefits of ACOT for their children 
than the disadvantages. The most common responses concerned the 
importance of computer literacy and observations of better attitude 
toward schooling and improved school performance. 

• Parents of children in primary and middle elementary grades 
(including the lower socioeconomic samples of upper 
elementary children at Site 3) were more likely to comment 
on ACOT's effects on children's attitude than were parents of 
upper elementary and secondary students. 

But some parents also expressed concerns for their children, focusing 
particularly on areas of the curriculum that parents felt had not 
received sufficient attention in 1988-89. 

ACOT parents' views of the effects of ACOT on themselves. 
Parents were less apt to discuss the benefits of ACOT for themselves 
(Table Z2). but there was some fairly common appreciation for the 
opportunity to use a computer at home. and. primarily at site 2, 
mention of positive effects on family interactions. Few pa-ents 
remarked on any disadvantages of the ACOT project for themselves. 
The relative infrequency of parents' comments about themselves 
may indicate that ACOT had not yet had widespread impact on 
parents and/or that parents' greatest concerns when responding to 
the questionnaire were with the impact of the computers on their 
children. 

Parents: Summary and Commentary . The uneven distribution 
of parent respondents across sites, grades, and groups limited our 
analyses particularly of comparison samples. In addition. 1988-89 
was the initial year for collection of these data, and we are therefore 
unable to use analyses of changes over time to help clarify the source 
of any ACOT -Comparison differences. These limitations, as for the 
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Teacher Questionnaire, limit sharply our ability to infer ACOT effects 
' on parents' views or home activities. 

The ACOT parents who participated in our questionnaires and 
interviews appeared to share fairly high aspirations for their 
children and to be only a somewhat active group at school. Parents 
were generally satisfied with their children's progress, although 
there were differences across sites which appeared to reflect parents' 
evolving views of education as their children advance with grade 
level. There was some limited evidence, given our small comparison 
sample, that ACOT children may continue school activities at home 
more often than Comparison children, a result mediated, we suspect, 
by the common presence of the computer in both contexts. Time 
spent by some ACOT children in social and physical play as well as 
TV watching may be less than comparison children -- but these 
results, if valid, may reflect the kind of child whose parents would 
apply for ACOT participation. ACOT children were the most frequent 
users of the home computer, but other family members and friends 
used it with some degree of frequency. There were interesting 
effects of site for these results: the computers were used by more 
persons for more time at Site 3 (lower socioeconomic level) v/here 
/ few homes would have a computer, and computers were used more 

^ exclusively by the ACOT child at Site 5 (working class, secondary) 

where students are assigned considerable homework and are 
encouraged to develop marketable computer skills. 

The computers appeared to be increasingly integrated in home 
activities, particularly by the ACOT child, with only occasional reports 
of fading interest for adults and other family members. Older 
children used the computer for more purposes than younger 
children, and most users used the computers for games, personal 
writing, practicing, and graphics. Adults at Sites 1 and 5 were the 
most different from one another in their patterns of use. reflecting 
perhaps the relatively highly educated sample at Site 1, and the less 
educated sample who were also parents of adolescents with 
homework and with needs, perhaps, to establish the computer as 
theirs' and that might discourage and even intimidate parents from 
efforts at computer literacy. The finding that many parents did 
report contexts in which their children had provided them assistance 
does suggest that a computer can be a context ^^.'hich may challenge 
established parent-child relationships in some households. 
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Parents were generally supportive of the ACOT project, and felt 
that it had benefited their children in any of several ways, including 
iheir children's knowledge of computers, attitudes toward learning, 
and achievement. But there were also concerns expressed about 
possible trade-offs in curriculum coverage. Some parents 
commented that the ACOT project had benefited them, particularly 
by providing them with the use of a computer. Few parents reported 
aay disadvantages for themselves. 

Implications of the Results 

In this section, we will discuss the implications of this report for 
understanding the effectiveness of ACOT and for planning future 
assessment efforts. 

Understanding the Effe ctiveness of ACOT 

Our approach to the study of ACOT effectiveness has been one 
of iriangulation. Recognizing the imperfections of existing measures 
and the scientific constraints on the real world laboratories in which 
ACOT is implemented, we are employing i strategy to assess progress 
based upon a' range of measures and multiple benchmarks. 
Comparisons of ACOT students' basic skills performance to nationally 
reported norms is one approach; comparisons of student progress 
and achiev3ment over time is another: comparisons of ACOT 
classrooms with demographically similar classrooms is still another 
information point; gathering data on classroom practices and parents' 
background characteristics to help explain and statistically adjust 
student outcomes is yet another. Our strategy is inherently 
developmental. We could not know at the outset the sets of 
measures most sensitive to technology-based interventions. We 
started with standard measures and now have extended our efforts 
to include the development of an expanded set of measurement tools 
to capture ACOT as it evolves. Thus, developing alternative measures 
of classroom process and students outcomes has been and continues 
to bs a key component in our assessment strategy. 

At this early stage, our study cannot provide firm conclusions 
about the effects of ACOT on student and othisr outcomes. Nor do we 
regard this information as central at this point. To recount some of 
the ongoing constraints of the study: ACOT is implemented in a 
relatively small number of classrooms, often only one at a particular 
grade level, dispersed in the original ACOT design over a diverse set 
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of school sites. ACOT effects are confounded by teachers influences, 
curriculum selection, school ambiance, and the characteristics of 
students, among other things. Participation of comparison classrooms 
in our study has been and continues to be problematic. Nonetheless, 
we believe some inferences are possible. 

The majority of our results thus far suggest that ACOT students 
have at least maintained their performance levels on standard 
measures of educational achievement in basic skills and have 
sustained positive attitudes as judged by measures addressing the 
traditional activities of schooling. Generally, the ACOT program 
appears at least as effective in promoting commonly measured 
student outcomes as the more typical instructional programs 
provided by the comparison sites, and in at least one site there are 
indications of advantage for ACOT students. 

How should such maintenance of the status quo be evaluated? 
We believe there are reasons to view it positively. First, it is clear 
that the ACOT environment differs enormously from traditional 
classroom practice and that it requires adapting to a host of new 
technological possibilities. For example, student time spent in 
learning word processing and other software is time that otherwise 
would be spent on traditional school subjects. Similarly, the time 
teachers need to acquire technology skills and familiarity with 
supportive software might result in less time spent in curriculum 
planning or instruction of certain skills. Some trial and error might 
be required to arrive at the software that is most appropriate and 
matched to individual and/or grade level needs or most effective for 
attaining given objectives, not to mention the time loss and 
frustration that result from occasional technical failures. Any of 
these short-term problems could result in less-than expected student 
academic growth or a temporary undermining of students' 
motivation and attitudes. These negative consequences on student 
outcome have not been observed. Second, and more significantly, the 
experience of ACOT itself appears to be resulting in significant new 
learning experiences for students and C'^eater attention to complex, 
higher level processing. Because more time in complex problem 
solving may translate into less time in basic skills instruction, some 
decrement in basic skills test performance also might be expected. 
From both these vantage points, then, maintenance of pre-ACOT 
performance levels could be viewed as an accomplishment. 
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Beyond maintaining performance on standard indicators. ACOT 
sites also show promising evidence of effectiveness in significant 
areas of education which are not well assessed by traditional 
measures: The non-standard measures which have been used by the 
study, assessing higher level outcomes and important dimensions of 
instructional process, reveal promising findings for future 
exploration. In two sites assessments of students' writing indicates 
substantial progress both over time and in comparison to similar 
groups of students. Our site visits and preliminary classroom 
observations, furthermore, suggest that ACOT is having positive and 
meaningful effects on the nature of instructional processes in 
participating classrooms. These instructional effects could be 
expected to produce significant outcomes, outcomes which again are 
not well represented by standard measures. For example, there are 
indications that some ACOT classrooms ieature greater emphasis on 
complex, higher level cognitive tasks, on student initiative and on 
cooperative group activities than do traditional classrooms. As we 
use our newly developed observation protocol to validate these 
impressions, we are simultaneously moving forward in developing 
new measures to better tap the likely outcomes of such changes. 

Student attitudes is another area where we have impressions 
of ACOT -related gains but .lave experienced some frustration with 
existing measures. The nationally normed measures we have used 
are the best available for group administration. Nonetheless, we and 
most measurement experts have serious reservations about the 
vaHdity and reliabi.Uty of currently existing scales dealing with such 
affective domains aj motivation, responsibility, and student self- 
concept. Further, the measures we have used focus on traditional 
school contexts and activities. Because the measures do not reflect 
the technology -rich environment of ACOT, interpretation of student 
responses is difficult. For example, do student responses represent 
reactions to traditional school activities as they have experienced 
them in the past or to technology-rich activities as they are 
experiencing them in ACOT? Again, observations in some classrooms 
suggest positive, affective impact — e.g., in engagement, commitment, 
pride in quality, but new approaches to measurement are required to 
document them. 

Future Directions for ACOT Assessment 

ACOT is a living labotatory in which the innovative possibilities 
of technology in schools are explored and refined. One of the 
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important strengths of the program is that it enables continuing 
experimentation with new possibilities, stimulating new goals, 
alternative instru ional processes, and new outcomes for students. 
Its evolutionary character presents a special assessment challenge: 
the valid assessment of ACOT requires both close-up interaction with 
sites to understand the nature of changes which are occurring and 
new tools which can capture the unique changes and complex 
outcomes which are being produced. The insufficiency of standard 
measures for assessing such change is one clear conclusion from our 
assessment efforts to date. Thus while we will continue to monitor 
and document the effects and impact of ACOT in the two longitudinal 
research center sites (LRC), we also will continue to devote 
substantial attention to creating new measurement tools which can 
adequately represent instructional transactions and complex project 
outcomes as they evolve. In the process. ACOT may well become 
not only a national model for technology use in instruction, but a 
national model for innovative assessment of both technological and 
instructional effects. 

Our innovative assessments draw on technology both as a 
stimulus to complex educational outcomes and as a medium for more 
effective measurement of those outcomes. Our recent and on-going 
efforts at developments of alternative assessments benefit from our 
experience as the national center for research on assessment and our 
understanding of technology in schools: 

A new mirror for the classroom. In 1988-89 we developed a 
new classroom observation instrument designed to capture the 
effects of technology on classroom instruction. The instrument 
focuses on the nature of the instructional tasks in which students are 
engaged, documenting cognitive, social, and affective dimensions of 
each task transpiring during given observation periods. Observation 
categories include subject area of instruction; social organization; the 
nature of materials which are given to students; the nature of the 
responses students are asked to produce; resources in use; the role of 
adults; and apparent affective responses to each task. Appendix AA 
contains the observation form developed (AAl). and the manuals for 
use at the elementary level (AA2) and the secondary level (AA2). 

The instrument is being used in 1989-90 at the two LRC ACOT 
sites. A primary function of our 1989-90 data collection is to begin 
the empirical documentation of commonly reported changes in the 
classroom practices of high access environments. Researchers and 
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educators seem to agree, for example, that computer use leads to 
more time on task, greater student motivation, more peer assistance, 
less directive teaching and more teacher facilitation, and more 
frequent group projects. However, prior to ACOT there has been 
little empirical documentation of these changes. 

A hvoermedia measure of know ledge representation. Among 
the two most visible movements in educational innovation are 
technology and assessment. Both represent the increasing call for 
educational reform - technology as an approach to improve 
instruction ard management; assessment as an instrument to 
measure progress. Yet, so far these processes have linked up in 
relatively limited ways, and we need to work toward constructing 
new relationships between instructional uses of computers and 
improved approaches to assessing student performance. The 
questions we must address include: How should assessment adapt to 
the realities of classrooms in which technology plays a major role? 
Can we move beyond the surface features of performance 
assessment? Can technology itself offer opportunities to assess 
learning with new levels of cognitive and instructional validity? 

There are few ways that educators are able to access students' 
understanding of complex phenomena, and almost all of them have 
serious drawbacks. Our hypermedia assessment project is an 
outgrowth of a long-term. Department of Education-funded study of 
the assessment of deep understanding through the use of extended 
student essays. Results i prior studies (Baker, 1988; Baker et al., 
1990) found that deep understanding of a subject was demonstrated 
by essays that were premise-driven, elaborated with text-specific 
and prior knowledge, and exhibited significant levels of 
interrelationships among facts, concepts and principles. Nonetheless, 
the quality of student verbal expression interacted with raters' 
ability to assess knowledge possessed. To investigate whether 
HyperCard representations could provide a more direct measure of 
students' understanding, the ACOT assessment study used HyperCard 
as a knowledge representation device for eleventh grade students. 
Students used a specially-designed stack to construct concept maps 
of Depression era history knowledge before writing essays on given 
topics. Scoring schemes for analyzing these HyperCard products are 
being developed. If successful, they may supplement or substitute 
for essay measures of deep understanding. 
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New measures of problem-solving. The interactive, responsive, 
and flexible character of computer use provides an inviting context 
for learning problem-solving skills. Many elementary level teachers 
have been attracted to instructional software packages that are 
designed to foster higher-level reasoning abilities, such as collection, 
organization, and appropriate use of information when making 
inferences. Because there is a conflicting body of evidence in 
psychological and educational research on the transfer of cognitive 
skills from one subject matter to another — and thus limited 
potential usefulness of isolated problem-solving practice, we felt it 
was important to document what kinds of gains, if any. could be 
demonstrated from use of problem-solving software. 

Based on 1988-89 field observations and teachers' descriptions 
of their problem -solving instruction, we designed tasks for solution 
both on-computer and off which were structurally and functionally 
similar to those practiced frequently at certain elementary ACOT 
sites. We are administering these tasks twice during 1989-90 (fall 
and spring) to a sample of ACOT and Comparison children who have 
been rated as either high or low in academic ability at one of two 
different grade levels. While we expect the results of the study to 
provide useful feedback to users of instructional problem-solving 
software, we also view the study as a model approach to the 
assessment of cognitive outcomes, one that stresses carefully 
motivated tasks based on documented classroom practices. 

Svstematic assessment of student portfolios. There is a 
dramatic movement in the field of educational measurement to go 
beyond standard, multiple choice tests to develop measures which 
better represent instructional outcomes and enable students to 
demonstrate skills. One major facet of this interest is the assessment 
of student portfolios. ACOT provides a unique opportunity to 
investigate how to systematically assess student portfolios, starting 
with those comprised of writing samples. In 1988-89 we collected 
from several sites examples of student writing as well as information 
from teachers about their writing curriculum. Based on this 
fieldwork and informed by knowledge of our colleagues' portfolio 
assessment efforts, we designed a portfolio evaluation procedure 
which we are implementing at the elementary level LRC site in 
1989-90. 

The portfolios will contain monthly samples of students' best 
writing in two genres that are emphasized at this site, as well as 
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samples of other projects that a student chooses to include. ACOT 
teachers are collaborating with UCLA's on-site researcher, in 
developing criteria for inclusion as well as procedures for 
documenting the instructional process that preceded the Hnal 
product. The resulting data set will contain: a series of writing 
samples to be evaluated on scales (to be developed) of writing 
competence: a set of student-selected projects that will inform our 
understanding of the students' views of their ACOT experience: and 
documentation of teachers* methods of writing instruction. The 
project offers opportunities for collaboration with Ohio State 
University researchers who will be exploring the bases for students' 
personal choices and the effects that opportunities for choosing, 
reviewing, and sharing portfolio selections can have on students' 
views of schooling and of themselves as learners. 

There is exciting potential for portfolios in all subject areas, in 
multisubject thematic areas, and in hypermedia formats. The ACOT 
sites will provide us unique contexts for exploring these possibilities. 
Although our initial effort will be devoted to collection and analysis 
of hard copies' of written work, we will be working toward 
computer-based portfolios. The range of product types that can be 
stored in hypermedia formats is virtually unlimited -- from written 
samples (scanned if handwritten), to art work (again scanned if not 
computer-based), to video, to audio, to HyperCard reports. Not only 
do we have access to technology and hypermedia software for 
storing and organizing students' work at ACOT sites, but many of the 
ACOT students* projects are themselves hypermedia products that 
are obviously most effectively displayed, reviewed, shared, and 
evaluated in a hypermedia format. 

Other non-co2nitive measures. Sensitive assessment of the 
effects of ACOT as well as other innovations in educational 
technology requires a varied measurement 'ool kit to capture a full 
range of high probability changes. Changes in students' attitudes — 
their persistence, independence, pride in work, among other 
attributes — are not only informally observed effects of ACOT but 
subjects of continuing national dialogue on this country's 
productivity and future competitiveness. De;'ising better measures 
of these constructs, likely grounded in observational, performance- 
based instruments, will be a continuing interest of the ACOT 
assessment component, an activity which will benefit from joint 
funding from the U.S. Department of Education. 
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A Ilninue Opportunity for Sensib le TechnoloBV ^.asessmeni 



We have repeatedly stressed that the assessment of ACOT 
requires continuing attention to new goals, new modes of 
instructional transaction, and new outcomes as they evolve. New 
tools are required to document and evaluate the complex effects of 
the ACOT; existing measures simply are insufficient to the task. 

ACOT presents an unparalleled opportunity to develop these 
new tools: it is an environment of innovation that explores the 
possibilities of technology in improving educational processes and 
outcomes. It is a laboratory also both for analyzing those processes 
and outcomes and for exploring the use of technology in the analysis 
process. In so being, ACOT can contribute to the improvement of 
instruction, the meaningful assessment of its effects, and to 
informed decision- making about effective innovation. 
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